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Summary 

Failure  of  military  personnel  to  report  their  status  to  their  commanders  may 
result  in  catastrophe  for  both  the  soldiers  and  their  units.  A  machine  capable  of 
detecting  cognitively  significant  events  in  its  user  could  prevent  potential  disaster  by 
signaling  to  commanders  that  a  soldier  is  xmder  high  stress.  This  project  seeks  to 
establish  that  cognitive  events  can  be  captured  in  an  autonomous  fashion  through  the 
use  of  an  eye-tracking  system.  To  demonstrate  feasibility,  a  laboratory  experiment  was 
conducted.  The  experiment  required  subjects  to  find  a  particular  person  hidden  in  a 
sequence  of  complex  images  that  contain  crowded  scenes  of  different  people 
performing  different  activities.  Project  tasks  included  creating  the  test  stimulus,  running 
test  subjects,  and  analyzing  the  captured  data.  This  analysis  indicates  that  a  pupil 
dilation  increase  during  a  period  of  prolonged  fixation  occurs  when  the  test  subject 
finds  the  target  person  in  the  stimulus.  Additional  testing  is  necessary  to  validate  this 
finding  in  a  more  realistic  setting,  but  this  study  represents  a  preliminary  step  in 
developing  a  machine  capable  of  autonomously  detecting  cognitive  events. 

Introduction 

The  interface  between  human  and  machine  has  classically  been  one-sided. 
Humans  command  machines  with  little  assistance  offered  by  the  machine  besides 
feedback  regarding  current  status  of  the  command  execution.  The  machine  has  been 
little  more  than  a  workhorse,  a  method  for  accomplishing  work  faster  and  more 
efficiently.  Over  the  past  decade,  the  idea  of  ubiquitous  con5)uting  has  arisen.  Under 
ubiquitous  computing,  machines  permeate  society,  and  the  interaction  between  human 
and  machine  becomes  more  collaborative  and  hidden  in  nature.  Machines  augment  how 
humans  perform  their  work  and  live  their  everyday  lives,  but  the  physical  presence  of 
the  machines  is  not  felt. 

Computational  power  has  been  a  major  impediment  to  the  proliferation  of 
ubiquitous  computing  devices.  Without  significant  processing  capabilities,  computers 
cannot  have  the  eyes  and  ears  they  need  to  successfully  partner  with  humans  in  a 
naturalistic  manner.  Fortunately,  recent  advances  in  both  computing  resources  and 
analytical  techniques  now  allow  the  first  steps  into  a  realistic  investigation  of  pervasive 
computing  techniques  to  occur. 

Augmentative  computing  can  be  particularly  helpful  in  areas  where  vigilance  of 
a  human  operator  is  necessary  for  successful  task  completion.  This  is  especially 
apparent  in  military  applications  where  military  personnel  must  report  their  status  to 
their  commanders.  Failure  to  do  so  may  be  catastrophic  for  both  the  individual  and  his 
or  her  unit.  A  machine  capable  of  detecting  cognitively  significant  events  would 
determine  when  military  persoimel  are  incapacitated  or  overloaded  due  to  a  high  stress 
level  associated  with  their  duties.  This  would  allow  commanders  to  assess  the  status  of 
all  of  their  troops  and  to  reallocate  resources  as  needed.  The  machine  could  also 
reiirforce  the  efforts  of  the  human,  possibly  streaming  additional  data  on  its  own  that  the 
human  should  be  reporting.  For  example,  consider  an  analyst  that  must  recognize 
friendly  versus  hostUe  aircraft.  If  many  aircraft  on  the  screen  appear,  the  analyst  may  be 
overloaded  as  he  or  she  examines  each  aircraft  and  may  not  report  his  or  her  findings  in 
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a  timely  manner.  Machines  capable  of  detecting  when  the  operator  recognizes  an 
aircraft  or  when  the  operator  is  engaged  in  significant  cognitive  activity  could  signal  the 
soldier’s  commander  that  recognized  aircraft  are  present  and  intervention  may  be 
necessary.  This  has  the  potential  to  revolutionize  combat  and  to  assist  America  as  it 
strives  to  maintain  the  superiority  of  its  armed  forces. 

This  project  seeks  to  establish  that  these  operationally  significant  cognitive 
events  can  be  captured  in  an  autonomous  fashion  through  the  use  of  an  eye-tracking 
system.  To  demonstrate  feasibility,  a  laboratory  experiment  was  conducted.  The 
following  report  describes  the  progress  made  in  this  experiment.  The  next  section 
discusses  the  eye-gaze  and  analytical  technology  employed  in  the  study.  The  Results 
and  Discussion  section  presents  the  test  stimuli  used  in  the  experiment  and  the  findings 
based  on  the  test  subject  data.  The  conclusion  presents  an  interpretation  of  the  data  with 
recommendation  for  further  work. 

Methods,  Assumptions,  and  Procedures 

The  Eye-gaze  Response  Interface  Computer  Aid  (ERICA)  is  a  unique  device 
developed  by  Eye  Response  Technologies  that  determines  where  its  user  is  looking  on  a 
computer  display.  ERICA  currently  serves  two  different  functions  [1]: 

1 .  Providing  individuals  with  disabilities,  such  as  quadriplegics,  with  eye- 
driven  computer  control.  The  user’s  eye  can  type  and  control  all  of  the 
mouse  functions  on  a  Windows  based  computer  thereby  enabling  access 
to  any  Windows  software. 

2.  Analyzing  where  people  look  as  they  view  static  images,  video  clips,  or 
interact  with  software.  These  users  include  researchers,  marketing  firms, 
and  software  usability  and  design  experts. 

Professor  Thomas  Hutchinson  conceived  ERICA  in  1 983  while  watching  a 
television  documentary  about  elephants.  He  noticed  that  the  photographers  used 
infrared  light  to  photograph  the  elephants.  Furthermore,  he  observed  a  relationship 
between  the  elephant’s  eye  movements  and  the  infrared  picture.  Hutchinson  reasoned 
that  a  computer  outfitted  with  an  infrared  camera  could  determine  the  gaze  position  of 
its  user  and  then  use  that  data  to  control  the  computer.  Research  focused  on 
implementing  the  system  was  successful.  [2]  outlines  the  advantages  of  having  an  eye- 
controlled  computer  and  describes  the  theory  behind  its  operation. 

The  ERICA  system  relies  on  infrared  light  to  determine  a  user’s  direction  of 
gaze.  An  infrared  light  emitting  diode  (LED)  resides  at  the  center  of  a  lens  that  is 
attached  to  a  video  camera  feeding  its  signal  to  a  computer.  This  LED  bathes  the  user’s 
face  in  infrared  light  at  880  nanometers,  a  wavelength  of  light  invisible  to  humans. 

When  the  light  from  this  LED  strikes  the  eye,  two  features,  the  glint  and  the  bright  eye, 
appear  in  the  camera  image,  as  shown  in  Figure  1.  The  glint  is  the  specular  reflection  of 
the  LED  from  the  front  surface  of  the  eye,  which  is  the  cornea.  Essentially,  a  specular 
reflection  is  the  intense  reflection  of  light  from  a  curved,  shiny  surface;  therefore,  the 
glint  appears  as  a  small,  bright  dot  to  the  camera.  The  bright  eye  is  formed  by  the 
absorption  and  reemission  of  the  infrared  light  by  the  retina  of  the  eye.  To  an  infrared 
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sensitive  camera,  this  makes  the  pupil  glow,  thereby  discriminating  it  from  the 
surrounding  iris  [1], 


Glint 


Figure  1.  Features  formed  when  infrared  light  strikes  the  eye. 


The  ERICA  system  locates  these  two  features  of  the  camera  image  and 
determines  where  the  user  is  looking  based  upon  the  vector  distance  between  the  two 
features,  as  shown  in  Figure  2  [1]. 


Directed  up  and  to  the  right  of  the  camera  Directed  up  and  to  the  left  of  the  camera 


Figure  2.  Glint  and  bright  eye  relationships. 


In  its  nearly  20-year  history,  over  300  students  have  performed  research  with  the 
device.  ERICA  developed  the  first  eye-controlled  computer  for  people  with  disabilities 
and  has  engaged  in  numerous  research  activities  concerning  the  analysis  of  eye 
movement  data.  These  activities  include  assessing  novice/expert  differences  in  tasks 
such  a  mammography  screening,  analyzing  user  interfaces,  determining  stress  level  and 
emotional  response  to  stimuli,  and  detecting  deception  by  test  subjects,  among  many 
other  areas  [3-15].  Many  experiments  have  shown  that  scan  path  and  pupil  dilation 
measures  can  indicate  differing  skill  levels  between  subjects  and  show  when  subjects 
experience  high  levels  of  cognitive  activity  or  stress  [3-5,  8]. 

ERICA  was  originally  created  to  assist  people  with  disabilities  by  providing 
them  with  a  means  to  communicate.  As  ERICA  developed,  another  potential  market  for 
the  system  became  apparent  -  the  system  could  facilitate  analysis  of  a  user’s  eye 
movements  as  well  as  determine  fluctuations  in  pupil  dilation.  Software  needed  to  be 
developed  to  allow  experimenters  to  construct  experiments  and  to  analyze  the  raw  data 
stream  of  gaze  positions  cuid  pupil  dilations  that  ERICA  emits  60  times  a  second. 
GazeTracker  is  the  application  that  resulted  from  this  development  [16]. 
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GazeTracker  offers  three  modes  -  image  analysis,  video  analysis,  and  software 
application  analysis.  GazeTracker's  image  analysis  mode  allows  researchers  to  study 
how  individuals  react  to  still  images  shown  on  the  computer  display.  It  outputs  fixation, 
scan  path,  and  pupil  dilation  measures.  Researchers  can  also  identify  regions  of  interest, 
called  LookZones,  that  give  more  in-depth  information  on  particular  areas  of  the  screen, 
such  as  percent  time  spent  in  the  area  or  number  of  times  the  area  was  observed. 
GazeTracker's  strength  rests  in  its  ability  to  facilitate  analysis  of  the  eye-tracking  data. 
Figure  3  depicts  GazeTracker  superimposing  a  test  subject’s  scan  path,  fixations,  and 
LookZones  on  the  stimulus.  The  size  of  the  fixation  reflects  the  fixation’s  duration.  The 
fixations  and  scan  path  have  adjustable  transparency  to  enable  viewing  of  the 
underlying  content.  Figure  4  shows  GazeTracker’s  3D  Analysis  method,  which  creates 
three-dimensional  views  of  the  stimulus  based  on  time  spent  in  the  regions  of  interest. 


4 


Current  Subject:  Time  in  I  nokZaiies 


Figure  4.  Three-dimensional  analysis  of  Figure  3. 

A  novel  feature  of  the  GazeTracker  software  is  its  ability  to  synchronize 
recording  of  eye-tracking  data  with  computer  state  data  from  any  Windows  application. 
The  gaze  position  on  the  computer  display  and  pupil  diameter  of  a  user  is  recorded  as 
they  operate  the  computer.  It  records  all  keystrokes  and  mouse  clicks  and  movements 
that  the  test  subject  performs  while  the  analysis  session  is  recording.  While  recording, 
both  the  eye-gaze  system  and  the  analysis  software  are  running  in  the  background  of  the 
computer,  passively  recording  the  actions  of  the  user  with  little  and  generally  unnoticed 
degradation  in  system  performance. 

GazeTracker’ s  video  analysis  records  how  individuals  react  to  video  files  shown 
on  the  computer.  The  software  knows  what  frame  of  the  video  file  is  shown  on  the 
screen,  so  it  can  associate  a  recorded  position  of  gaze  to  the  frame  shown.  Furthermore, 
the  LookZones  may  be  specified  to  change  size,  shape,  and  position  throughout  the 
video.  This  allows  a  LookZone  to  be  placed  over  a  moving  object,  such  as  the  aircraft  in 
the  AVI  files  described  above,  and  the  software  would  still  provide  accurate  measures 
relating  to  the  number  of  times  the  LookZone  is  observed,  time  spent  in  the  LookZone, 
or  pupil  dilation  while  the  zone  is  observed.  This  feature  is  a  critical  and  unique 
component  of  the  GazeTracker  system  and  greatly  simplifies  analysis  in  the  experiment 
described  in  the  original  proposal. 

In  the  original  experiment,  the  stimulus  shown  to  each  test  subject  was  to  be 
comprised  of  audio-video  interleave  (AVI)  files.  The  system  would  measure  the 
subject’s  eye  response  as  they  view  each  file.  Each  file  would  show  vehicles  navigating 
a  scene. 

However,  after  discussion  with  the  technical  monitor.  Dr.  Allan  Steinh£irdt,  the 
decision  was  made  to  employ  sequences  of  images  instead  of  AVI  files.  This  would 


5 


simplify  the  experiment  in  an  effort  to  uncover  the  basic  impact  of  an  image  recognition 
effect  on  pupil  diameter  and  fixations  metrics. 

The  initial  experiment  was  divided  into  two  phases.  The  first  phase  tested  object 
recognition,  and  the  second  phase  tested  person  recognition.  In  each  phase,  the  subject 
was  presented  with  a  random  sequence  of  images.  Each  image  was  divided  into  nine 
blocks.  Each  block  had  a  different  graphic.  Before  each  phase,  the  test  subject  was 
shown  a  graphic  to  recognize.  During  the  experiment,  whenever  the  test  subject  saw  the 
graphic  to  be  recognized,  he  or  she  pressed  the  spacebar.  When  they  had  finished 
examining  all  blocks  in  an  image,  they  said,  “Done,”  and  the  experimenter  then 
instructed  the  GazeTracker  software  to  present  the  next  image  by  clicking  the  mouse 
button.  Each  image  had  between  zero  and  nine  graphics  to  be  recognized.  Each  phase 
had  twelve  images.  Thirty  valid  recognitions  should  have  occurred  in  each  phase. 
Sample  images  from  the  two  phases  are  shown  in  Figure  5  and  Figure  6.  The  tank 
graphic  should  be  recognized  in  Phase  I.  The  Saddam  graphic  should  be  recognized  in 
Phase  II. 


j 
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Figure  5.  Sample  object  recognition  picture. 
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Figure  6.  Sample  fece  recognition  picture. 


After  creating  the  stimulus,  testing  started.  Eleven  subjects  were  tested. 
Preliminary  data  analysis  was  conducted  on  these  subjects. 

This  analysis  focused  on  visually  inspecting  trends  in  pupil  diameter.  Trends 
seem  to  be  apparent  only  on  the  first  images  observed  in  each  phase  for  each  subject. 
Specifically,  the  pupil  diameter  seems  to  decrease  and  then  increase  approximately  one 
to  two  seconds  after  the  image  appeared.  On  the  remaining  images,  pupU  dilation  does 
not  seem  to  consistently  change  in  such  a  significant  manner.  Interviews  with  the  test 
subjects  revealed  that  most  subjects  felt  the  test  was  easy  after  they  had  performed  the 
task  on  the  first  image.  They  could  even  rely  on  peripheral  vision  to  accomplish 
identification.  This  implies  that  the  pupil  dilation  change  is  related  more  to 
concentration  or  mental  workload  effects  than  to  recognition  effects.  Figure  7  and 
Figure  8  show  example  pupil  graphs  for  a  test  subject  observing  the  first  images 
displayed  in  both  phases. 


Figure  7.  Sample  object  recognition  phase  pupil  graph. 


Figure  8.  Sample  face  recognition  phase  pupil  graph. 


After  presenting  these  preliminary  results  to  Dr.  Steinhardt,  it  was  decided  to 
include  a  more  complex  recognition  task  in  the  experiment.  Different  stimuli  were  then 
examined.  Specifically,  AVI  files  of  crowd  scenes,  AVI  files  of  shapes  morphing  from 
one  design  to  another,  and  intelligence  photographs  from  different  US  military  actions 
were  investigated.  Each  design  was  eliminated  because  of  either  difficulty  in  generating 
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an  appropriate  test  stimulus  (the  AVI  files)  or  due  to  specialized  knowledge  being 
needed  by  the  test  subjects  (intelligence  photographs).  Consequently,  the  final 
experiment  used  pictures  from  the  Where ’s  Waldo  series  of  children’s  books  [17-18]. 
These  books  require  its  reader  to  examine  an  extremely  detailed  crowd  scene  of 
different  people  performing  many  different  tasks.  The  goal  is  for  the  reader  to  find  the 
Waldo  character.  Each  scene  contains  many  elements  that  resemble  the  Waldo 
character,  but  the  Waldo  character  appears  only  once.  Figure  9  and  Figure  10  shows  the 
Waldo  character  and  a  sample  image.  Six  images  from  two  different  bodks  were 
scanned  into  the  computer  and  used  in  GazeTracker’s  Image  Analysis  mode.  Each  test 
subject  observed  an  image  until  he  or  she  found  Waldo.  The  subject  then  pressed  the 
spacebar  on  the  keyboard,  and  the  system  advanced  to  the  next  image. 


Figure  9.  Waldo  character  [17]. 


Figure  10.  Sample  Waldo  scene  with  Waldo  circled  (Waldo  was  not  circled  during  the  test)  [17]. 

Results  and  Discussion 

Ten  subjects  were  tested  in  this  study.  Due  to  time  constraints,  rigorous 
statistical  analysis  of  the  collected  data  could  not  be  performed;  however,  several 
general  trends  appeared  that  merit  further  investigation. 

Analysis  focused  on  images  for  which  the  test  subjects  exhibited  difficulty  in 
finding  Waldo  as  reflected  by  the  total  time  spent  examining  the  image.  Subject 
observation  times  were  clustered  below  ten  seconds  and  above  fifteen  seconds,  so  only 
images  where  subjects  observed  the  picture  for  more  than  fifteen  seconds  were 
analyzed.  Seven  of  the  ten  subjects  observed  one  or  more  image  for  more  than  fifteen 
seconds.  A  total  of  eleven  images  were  observed  for  more  than  fifteen  seconds. 

The  primary  finding  of  this  study  was  that  fixations  with  an  increase  in  pupil 
dilation  and  a  duration  greater  than  fifty  percent  of  the  mean  fixation  duration  occurred 
when  a  subject  observed  Waldo.  The  data  showed  other  prolonged  fixations  and  other 
increases  in  pupil  dilation,  but  a  prolonged  fixation  combined  with  an  increase  in  pupil 
dilation  did  not  occur  unless  the  subject  was  observing  Waldo.  Figure  1 1  and  Figure  12 
show  an  example  of  this  trend.  The  size  of  the  fixation  indicates  its  duration. 

Figure  11.  Scan  path  showing  a  Waldo  search.  Waldo  is  under  Fixations  98  and  99  in  the  left  image  and 

146  in  the  right. 


Figure  12.  Pupil  dilation  graphs  for  the  Waldo  search.  Note  the  increase  in  pupil  dilation  at  the  end  of  the 

search  when  Waldo  is  found. 


Figure  13  shows  a  subject  who  identified  that  he  found  Waldo;  however,  he  did 
not  find  the  true  Waldo.  As  shown  in  Figure  13  and  Figure  14,  there  is  a  large  fixation 
at  the  incorrect  Waldo,  but  there  is  not  a  corresponding  increase  in  pupil  dilation  at  the 
end  of  the  search. 
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Figure  13,  Scan  path  showing  a  subject  mis-identifying  Waldo  (Fixation  59).  The  correct  Waldo  is 

circled  red. 


Figure  14.  Pupil  dilation  graph  for  the  above  visual  search.  There  is  no  increase  in  pupil  dilation  at  the 

end  of  the  search. 


Conclusions 

The  trends  described  above  appeared  in  the  eleven  images  meeting  the  selection 
criteria  for  total  search  duration.  This  preliminary  study  shows  promise  in  finding  a 
metric  that  can  indicate  when  image  recognition  occurs;  however,  several  additional 
studies  will  be  needed. 

First,  another  study  that  requires  the  test  subject  to  find  two  people  in  the  image 
should  be  conducted.  This  would  allow  study  into  how  the  pupil  changes  after 
recognition  occurs.  Fortunately,  the  Waldo  series  of  children’s  books  have  images 
where  two  people  need  to  be  found  -  a  Wizard  and  Waldo.  The  same  protocol  described 
above  could  be  used  for  these  images.  The  subject  would  press  the  spacebar  each  time 
he  or  she  finds  the  target,  and  the  image  would  change  after  two  key  presses. 

After  conducting  another  study  and  finding  if  the  same  recognition  effect  holds, 
a  real-time  method  to  predict  when  Waldo  was  found  with  the  collected  data  could  be 
developed.  The  next  study  would  not  have  the  test  subject  signal  when  they  had  found 
Waldo;  the  image  would  automatically  advance  when  the  metrics  related  to  Waldo 
recognition  occurred. 

Assuming  the  above  tasks  were  successful,  the  metrics  could  then  be  studied  in  a 
real-world  military  situation,  such  as  when  intelligence  analysts  examine  incoming 
image  data  and  must  make  quick  decisions  concerning  the  nature  of  a  possible  threat.  If 
the  metrics  apply  to  a  real-world  situation,  this  would  allow  a  system  to  act  as  an 
assistant  to  an  image  analyst.  When  the  analyst  fails  to  consciously  recognize  an  object, 
the  system  could  signal  to  the  analyst  that  they  need  to  examine  the  object  in  more 
detail.  This  would  be  one  of  the  first  examples  of  a  machine  capable  of  autonomously 
detecting  cognitive  events. 
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Figure  12.  Pupil  dilation  graphs  for  the  Waldo  search.  Note  the  increase  in  pupil  dilation  at  the  end  of  the 

search  when  Waldo  is  found. 


Figure  13  shows  a  subject  who  identified  that  he  found  Waldo;  however,  he  did 
not  find  the  true  Waldo.  As  shown  in  Figure  13  and  Figure  14,  there  is  a  large  fixation 
at  the  incorrect  Waldo,  but  there  is  not  a  corresponding  increase  in  pupil  dilation  at  the 
end  of  the  search. 


Figure  13.  Scan  path  showing  a  subject  mis-identifying  Waldo  (Fixation  59).  The  correct  Waldo  is 

circled  red. 
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Figure  14.  Pupil  dilation  graph  for  the  above  visual  search.  There  is  no  increase  in  pupil  dilation  at  the 

end  of  the  search. 


Conclusions 

The  trends  described  above  appeared  in  the  eleven  images  meeting  the  selection 
criteria  for  total  search  duration.  This  preliminary  study  shows  promise  in  finding  a 
metric  that  can  indicate  when  image  recognition  occurs;  however,  several  additional 
studies  will  be  needed. 

First,  another  study  that  requires  the  test  subject  to  find  two  people  in  the  image 
should  be  conducted.  This  would  allow  study  into  how  the  pupil  changes  after 
recognition  occurs.  Fortunately,  the  Waldo  series  of  children’s  books  have  images 
where  two  people  need  to  be  found  -  a  Wizard  and  Waldo.  The  same  protocol  described 
above  could  be  used  for  these  images.  The  subject  would  press  the  spacebar  each  time 
he  or  she  finds  the  target,  and  the  image  would  change  after  two  key  presses. 

After  conducting  another  study  and  finding  if  the  same  recognition  effect  holds, 
a  real-time  method  to  predict  when  Waldo  was  found  with  the  collected  data  could  be 
developed.  The  next  study  would  not  have  the  test  subject  signal  when  they  had  found 
Waldo;  the  image  would  automatically  advance  when  the  metrics  related  to  Waldo 
recognition  occurred. 

Assuming  the  above  tasks  were  successful,  the  metrics  could  then  be  studied  in  a 
real-world  military  situation,  such  as  when  intelligence  analysts  examine  incoming 
image  data  and  must  make  quick  decisions  concerning  the  nature  of  a  possible  threat.  If 
the  metrics  apply  to  a  real-world  situation,  this  would  allow  a  system  to  act  as  an 
assistant  to  an  image  analyst.  When  the  analyst  fails  to  consciously  recognize  an  object, 
the  system  could  signal  to  the  analyst  that  they  need  to  examine  the  object  in  more 
detail.  This  would  be  one  of  the  first  examples  of  a  machine  capable  of  autonomously 
detecting  cognitive  events. 
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